Since November 2013 I have been using Replicant on my Samsung S3 as an alternative OS. The experience has been good for everyday use. The limits (due to non-free software components) compared to a normal S3 (running vendor ROM or CyanogenMod) is lack of GPS/wifi/bluetooth/NFC/frontcamera functionality although it is easy to get some of that working again, including GPS, which is nice for my geocaching hobby. The Replicant software is stable for being an Android platform; better than my Nexus 7 (2nd generation) tablet which I got around the same time that runs an unmodified version of Android. The S3 has crashed around ten times in these four months. I ve lost track of the number of N7 crashes, especially after the upgrade to Android 4.4. I use the N7 significantly less than the S3, reinforcing my impression that Replicant is a stable Android. I have not had any other problem that I couldn t explain, and have rarely had to reboot the device. The Replicant project recently released version 4.2 and while I don t expect the release to resolve any problem for me, I decided it was time to upgrade and learn something new. I initially tried the official ROM images, and later migrated to using my own build of the software (for no particular reason other than that I could). Before the installation, I wanted to have a full backup of the phone to avoid losing data. I use SMS Backup+ to keep a backup of my call log, SMS and MMS on my own IMAP server. I use oandbackup to take a backup of all software and settings on the phone. I use DAVDroid for my contacts and calendar (using a Radicale server), and reluctantly still use aCal in order to access my Google Calendar (because Google does not implement RFC 5397 properly so it doesn t work with DAVDroid). Alas all that software is not sufficient for backup purposes, for example photos are still not copied elsewhere. In order to have a complete backup of the phone, I m using rsync over the android debug bridge (adb). More precisely, I connect the phone using a USB cable, push a rsyncd configuration file, start the rsync daemon on the phone, forward the TCP/IP port, and then launch rsync locally. The following commands are used: jas@latte:~$ cat rsyncd.conf
address 127.0.0.1
uid = root
gid = root
[root]
path = /
jas@latte:~$ adb push rsyncd.conf /extSdCard/rsyncd.conf
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
0 KB/s (57 bytes in 0.059s)
jas@latte:~$ adb root
jas@latte:~$ adb shell rsync --daemon --no-detach --config=/extSdCard/rsyncd.conf &
jas@latte:~$ adb forward tcp:6010 tcp:873
jas@latte:~$ sudo rsync -av --delete --exclude /dev --exclude /acct --exclude /sys --exclude /proc rsync://localhost:6010/root/ /root/s3-bup/
...
Now feeling safe that I would not lose any data, I remove the SIM card from my phone (to avoid having calls, SMS or cell data interrupt during the installation) and follow the Replicant Samsung S3 installation documentation. Installation was straightforward. I booted up the newly installed ROM and familiarized myself with it. My first reaction was that the graphics felt a bit slower compared to Replicant 4.0, but it is hard to tell for certain. After installation, I took a quick rsync backup of the freshly installed phone, to have a starting point for future backups. Since my IMAP and CardDav/CalDav servers use certificates signed by CACert I first had to install the CACert trust anchors, to get SMS Backup+ and DAVDroid to connect. For some reason it was not sufficient to add only the root CACert certificate, so I had to add the intermediate CA cert as well. To load the certs, I invoke the following commands, selecting Install from SD Card when the menu is invoked (twice). adb push root.crt /sdcard/
adb shell am start -n "com.android.settings/.Settings\"\$\"SecuritySettingsActivity"
adb push class3.crt /sdcard/
adb shell am start -n "com.android.settings/.Settings\"\$\"SecuritySettingsActivity"
I restore apps with oandbackup, and I select a set of important apps that I want restored with settings preserved, including aCal, K9, Xabber, c:geo, OsmAnd~, NewsBlur, Google Authenticator. I install SMS Backup+ from FDroid separately and configure it, SMS Backup+ doesn t seem to want to restore anything if the app was restored with settings using oandbackup. I install and configure the DAVdroid account with the server URL, and watch it populate my address book and calendar with information. After organizing the icons on the launcher screen, and changing the wallpaper, I m up and running with Replicant 4.2. This upgrade effort took me around two evenings to complete, with around half of the time consumed by exploring different ways to do the rsync backup before I settled on the rsync daemon approach. Compared to the last time, when I spent almost two weeks researching various options and preparing for the install, this felt like a swift process.
I spent some time researching how to get the various non-free components running. This is of course sub-optimal, and the Replicant project does not endorse non-free software. Alas there aren t any devices out there that meets my requirements and use only free software. Personally, I feel using a free core OS like Replicant and then adding some non-free components back is a better approach than using CyanogenMod directly, or (horror) the stock ROM. Even better is of course to not add these components back, but you have to decide for yourselves which trade-offs you want to make. The Replicant wiki has a somewhat outdated page on Samsung S3 firmware. Below are my notes for each component, which applies to Replicant 4.2 0001. You need to first prepare your device a bit using these commands, and it is a good idea to reboot the device after installing the files. adb root
adb shell mount -o rw,remount /system
adb shell mkdir /system/vendor/firmware
adb shell chmod 755 /system/vendor/firmware
GPS: The required files are the same as for Replicant 4.0, and using the files from CyanogenMod 10.1.3 works fine. The following commands load them onto the device. Note that this will load code that will execute on your main CPU which is particularly bothersome. There seems to exist a number of different versions of these files, CyanogenMod have the same gpsd and gps.exynos4.so in version 10.1.3 and 10.2 but the libsecril-client.so differs between 10.1.3 and 10.2. All files differ from the files I got with my stock Samsung ROM on this device (MD5 checksums in my previous blog). I have not investigated how these versions differs or which of them should be recommended. I use the files from CyanogenMod 10.1.3 because it matches the Android version and because the files are easily available. adb push cm-10.1.3-i9300/system/bin/gpsd /system/bin/gpsd
adb shell chmod 755 /system/bin/gpsd
adb push cm-10.1.3-i9300/system/lib/hw/gps.exynos4.so /system/lib/hw/gps.exynos4.so
adb push cm-10.1.3-i9300/system/lib/libsecril-client.so /system/lib/libsecril-client.so
adb shell chmod 644 /system/lib/hw/gps.exynos4.so /system/lib/libsecril-client.so
Bluetooth: Only one file has to be installed, apparently firmware loaded onto the Bluetooth chip. Cyanogenmod 10.1.3 and 10.2 contains identical files, which has a string in it BCM4334B0 37.4MHz Class1.5 Samsung D2 . The file I got with my stock ROM has a string in it BCM4334B0 37.4MHz Class1.5 Samsung M0 . I don t know the difference, although I have seen that D2 sometimes refers to the US version of a Samsung device. My device is the international version, but it seems to work anyway. adb push cm-10.1.3-i9300/system/bin/bcm4334.hcd /system/vendor/firmware/bcm4334.hcd
adb shell chmod 644 /system/vendor/firmware/bcm4334.hcd
Front Camera: Two files has to be installed, apparently firmware loaded onto the Camera chip. CyanogenMod 10.1.3 and 10.2 contains identical files, which has a string in it [E4412 520-2012/08/30 17:35:56]OABH30 . The file I got with my stock ROM has a string in it [E4412 533-2012/10/06 14:38:46]OABJ06 . I don t know the difference. adb push cm-10.1.3-i9300/system/vendor/firmware/fimc_is_fw.bin /system/vendor/firmware/fimc_is_fw.bin
adb push cm-10.1.3-i9300/system/vendor/firmware/setfile.bin /system/vendor/firmware/setfile.bin
adb shell chmod 644 /system/vendor/firmware/fimc_is_fw.bin /system/vendor/firmware/setfile.bin
NFC: I m happy that I got NFC to work, this was one of my main issues with Replicant 4.0 (see my earlier blog post). Only one file is needed, however CyanogenMod does not seem to distribute it so you have to get it from your stock ROM or elsewhere. The md5 of the file I have is b9364ba59de1947d4588f588229bae20 (and no I will not send it to you). I have tested it with the YubiKey NEO and the Yubico Authenticator app. adb push clockworkmod/blobs/ee6/7188ca465cf01dd355a92685a42361e113f886ef44e96d371fdaebf57acae /system/vendor/firmware/libpn544_fw.so
adb shell chmod 644 /system/vendor/firmware/libpn544_fw.so
Wifi: I haven t gotten wifi to work, although I have not tried very hard. Loading the CyanogenMod firmwares makes my device find wireless networks, but when I try to authenticate (WPA-PSK2), I get failures. Possibly some other files has to be loaded as well. Update: This blog post has been updated since initial posting to use rsync over adb instead of USB tethering, and to mention that I got the ROM building to work.

I'm not an avid Kickstarter follower (if I were, I'd have gone broke long ago). I tend to wind up backing campaigns that come to my attention via other means (in other words, they're in the process of going viral). That said, I've backed plenty of campaigns over the years, and so I'd like to think I have a little bit of familiarity with how they usually operate. When Ozobot came to my attention, I found it unusual, because they were pre-promoting their Kickstarter campaign before it opened. To me, this looked like a case of them trying to build hype prior to the campaign opening, which was a new one to me. The whole thing seemed incredibly slick, and I was surprised they were "only" seeking $100K. The product looked like it'd be something cool for Zoe to play with, so I decided to back it anyway. Then all the updates started flowing in about how well it was being received at various trade shows and whatnot. Yet the amount of dollars flowing into the Kickstarter campaign didn't seem to be reflecting the external hype. I was watching the campaign's dashboard with great interest, because as time marched on, it was looking more and more likely that it wasn't going to make its funding target. This seemed highly unusual to me, given the slickness of the product and purported external interest in it. And then they pulled the plug on the campaign. Purportedly because they were pursuing equity funding instead. They admitted they'd also read the writing on the wall and it was unlikely they were going to make their funding target. I haven't followed other campaigns to see how much of a last minute funding "pop" they have. Usually I've found they've closed at many multiples of their original target, and hit their target well in advance of their deadline, when they're ridiculously popular. My interpretation of Ozobot's campaign, from a funding perspective, is that Kickstarters gave it a big fat "MEH", which surprised me somewhat. Then the question comes up: was the Kickstarter campaign a ruse all along? Was it just a new way of pitching for venture capital? The videos seemed pretty slick. The product seemed already complete, and $100K didn't seem like enough to take it to manufacturing. It'll be interesting to see what becomes of Ozobot now.

Another post about the Valve/Collabora free games thing. This time, the bad bit people trying to scam free games from us. Before I start, I want to make one thing clear there are people who have requested keys who don t meet the criteria, but are honest and legitimate in their requests. This blogspam is not about them at all. If you re in that category, you re not being complained about. So. Some numbers. At time of writing, I ve assigned keys to 279 Debian Developers or Debian Maintainers almost 25% of the total eligible pool of about 1200. I ve denied 22 requests. Of these 10 were polite requests from people who didn t meet the conditions stated (e.g. Ubuntu developers rather than Debian). These folks weren t at all a problem for us, and I explained politely that they didn t meet the terms we had agreed at the time with Valve. No problem at all with those folks. Next, we have the chancers, 7 of them, who pretended to be eligible when they clearly weren t. For example, two people sent me signed requests pointing to their entry on the Debian New Maintainers page when challenged over the key not being in the keyring. The NM page showed that they had registered as non-uploading Debian Contributors a couple of hours previously. A few just claimed I am a DD, here is my signature when they weren t DDs at all. Those requests were also binned. And then we move onto the final category. These people tried identity theft, but did a terrible job of it. There were 5 people in this category:

From: Xxxxxxxx Xxxxxx <xxxxxxxx.xxxxxx@ieee.org>
Subject: free subscription to Debian Developer
8217 A205 5E57 043B 2883 054E 7F55 BB12 A40F 862E

This is not a signature, it s a fingerprint. Amusingly, it s not the fingerprint for the person who sent my mail, but that of Neil McGovern a co-worker at Collabora. Neil assured me he knew how to sign a mail properly, so I shitcanned that entry.

From: "Xxxxx, Xxxxxxxxx" <x.xxxxx@bbw-bremen.de>
Subject: Incoming!
Hey dude,
I want to have the redemption code you are offering for the Valve Games
mQGiBEVhrscRBAD4M5+qxhZUD67PIz0JeoJ0vB0hsLE6QPV144PLjLZOzHbl4H3N
...snip...
Lz8An1TEmmq7fltTpQ+Y1oWhnE8WhVeQAKCzh3MBoNd4AIGHcVDzv0N0k+bKZQ=3D=3D
=3Du/4R

Wat? Learn to GPG!

From: Xxxxxx-Xxxx Le Xxxxxxx Xxxx <xx.xxxxxxxxx@gmail.com>
Subject: pass steam
Hey me voila
Merci beaucoup
valve
2069 1DFC C2C9 8C47 9529 84EE 0001 8C22 381A 7594

Like the first one, a fingerprint. This one is for S bastien Villemot. Don t scammers know how to GPG sign?

From: "Xxxxxxxxx Xxxxxxx" <xxxxxxxx@web.de>
Subject: thanks /DD/Steam gifts us finally something back
0x6864730DF095E5E4

Yet again, a fingerprint. This one is for Marco Nenciarini. I found this request particularly offensive due to the subject line the haughty tone from an identity thief struck me as astonishingly impertinent. Still, when will scammers learn to GPG?

From: Sven Hoexter <svenhoexter@gmail.com>
Subject: Valve produced games
I'm would like to get the valve produced games
My keyring: 0xA6DC24D9DA2493D1 Sven Hoexter <hoexter> sig:6

Easily the best scam effort, since this is the only one which both a) registered an email address under the name of a DD, and b) used a fingerprint which actually corresponds to that human. Sadly for the scammer, I m a suspicious kind of person, so my instinct was to verify the claim via IRC.

31-01-2014 16:52:48 > directhex: Hoaxter, have you started using gmail without updating your GPG key? (note: significantly more likely is someone trying to steal your identity a little to steal valve keys from collabora)
31-01-2014 16:54:51 < Hoaxter!~sh@duckpond6.stormbind.net: directhex: I do not use any Google services and did not change my key

So yeah. Nice try, scammer. I m not listing, in all of this, the mails which Neil received from people who didn t actually read his mail to d-d-a. I m also not listing a story which I ve only heard second ha actually no, this one is too good not to share. Someone went onto db.debian.org, did a search for every DD in France, and emailed every Jabber JID (since they look like email addresses) asking them to forward unwanted keys. All in all, the number of evildoers is quite low, relative to the number of legitimate claims 12 baddies to 279 legitimate keys issued. But still, this is why the whole key issuing thing has been taking me so long and why I have the convoluted signature-based validation system in place. Enjoy your keys, all 279 of you (or more by the time some of you read this). The offer has no explicit expiry on it Valve will keep issuing keys as long as there is reason to, and Collabora will continue to administer their allocation as long as they remain our clients. It s a joint gift to the community thousands of dollars worth of games from Valve, and a significant amount of my time to administer them from Collabora.

I'm not a strong swimmer, and never have been. I was never confident in the water as a child. In my teenage years, I had some skin issues that I was very self-conscious about. Going to a boy's school where I was already bullied, I didn't want to add material, so I avoided swimming. To this day, I can barely swim 50 metres freestyle. To think I once had aspirations of getting into the police, which required a bronze medallion. As as result of my atrocious swimming abilities, it's been very important to me that Zoe be able to swim well. Purely for her own personal safety if nothing more. She's been doing "swim classes" of various sorts since she's been 6 months old. In the US, they were group parent/child water familiarisation classes. When we moved back to Australia and turned three, she was eligible to start doing small-group instructor-led classes with Hampton Swim School. It felt like such an milestone just to no longer have to be in the pool with Zoe. I was initially quite skeptical of Hampton's methods, compared with what I'd observed at the Betty Wright Swim School (which subsequently had to close due to aging facilities). Having seen how well Zoe's swimming has progressed in 3 quarters of classes is just amazing. The last few times I've taken Zoe to a public pool for some non-class swim time, her confidence in the water has been fantastic. I've bought some "sinkies", toys that sink to the bottom, and she'll happily dive down to the bottom to retrieve them (with assistance from me getting down there). Today, to escape the 40 plus degree heat, I took her to the Sleeman Sports Centre, because they have a really cool slide, and the whole thing's indoors. Zoe was confidently swimming between me and the side of the pool in water that was over her head, repeatedly. So to have her go from being fine in the water as long as she was being held, to actively asking me to go further back from the wall, in under a year has been really excellent progress. She's also "diving" in off the side (it's more like a bad belly flop, but she has brilliant form at the start, with her hands over her head). We got to the pool today around 10:30am, and it turned out the slide didn't open until 1pm, so we decided to stick around for lunch and wait until the slide opened. To kill time after lunch, we went to the 50 metre pool, where some other kids were taking turns to jump in off the starting blocks. Fearless Zoe wanted to have a go too. This was seriously deep water. It was really inspiring watching this tiny little girl climb up onto the starting blocks of the 1982 Commonwealth Games 50 metre pool and "dive" into the water, flounder over to me, who was furiously treading water in the middle of the lane, and then flounder over to the side, climb out unassisted and do it all over again. It was a seriously "proud father" moment, while I was trying not to drown myself in the middle of the lane. I wish I'd had someone there with a camera to capture the moment. I have great hopes for the future of her swimming, and I think she's ready to go up a level in her swim class.

Review: Why We Fail, by Victor Lombardi

Publisher:	Rosenfeld Media
Copyright:	2013
ISBN:	1-933820-17-9
Format:	Kindle
Pages:	248

Subtitled Learning from Experience Design Failures, Why We Fail is a case study book. It defines experience design, discusses the usefulness of analyzing failure, and then walks through ten marketplace failures attributed to failures of user experience (as opposed to engineering or user interface) and tries to draw some conclusions about those failures and how they could have been prevented. Lombardi then presents a more general (if fairly simple) framework for iterative experience design, along with some discussion of how that approach could have helped in these cases. Hardware and software designs can work properly but still fail with users and customers because the overall experience isn't right. Sometimes this is because something else got there first with more market share and people got used to it, sometimes it's because a system is too complicated, sometimes it's too simple, sometimes it's just uninspiring. Most of the news media coverage and many of the blog posts on this topic look to inspirational examples of successes. Lombardi points out here that successes can be accidental and failures are often more informative, a concept already common in many more mature areas of engineering. His hope, with Why We Fail, is to push a broader industry practice of taking apart and analyzing the failures. User experience design is still in its infancy, since many of the complex interactions now possible in modern hardware and software have just recently crossed into levels of complexity that qualify as experiences rather than interactions, and could benefit from more rigor. The concept sounds great. The approach in Why We Fail is at least interesting; Lombardi tells good stories, and I think they do tell some useful lessons. Drawing general principles is harder, which is always the weakness of case-study books. Lombardi's own attempt doesn't go far beyond a familiar mantra of the scientific method, small and responsive teams, cross-functional teams with flexibility to address a whole range of technical and business issues, and honest and forthright analysis of and iteration after failure. All of this is doubtless good advice; none of it should surprise anyone with even a smattering of engineering background or familiarity with current software design philosophy. The heart of the book is in the stories, and I did like that Lombardi captured a wide range of different designs. Included are embedded systems (BMW's iDrive), traditional software packages, web services of various types, and even a protocol (OAuth). Some were vaguely familiar and some were not familiar at all, although I don't follow UI and UX discussions to any large extent. In each case, Lombardi gives a short but effective introduction to the problem space and the product, walks through the design choices, and then talks about the failure. Usefully, he doesn't stop with the failure but continues the story through the reaction of the company to the failure and any subsequent actions they took, which in some cases contain their own useful lessons. All of this, I should note, is a bit far afield of my own area of expertise (architectural building blocks and sub-surface components without as much direct interaction with the user), so I'm not the best person to judge the quality and originality of the analysis. But I liked Lombardi's approach and his analysis of multiple possible failure causes, and all of his conclusions seemed quite plausible and sensible given the case studies as described. I'm best-qualified to judge the chapter on OAuth, since I've worked extensively on web authentication systems, and his analysis of the challenges and user experiences in that case closely match my own. I didn't notice any significant errors, which is hopefully a good sign for the rest of the book. As one might expect from a short book about a complex field aimed mostly at starting a conversation, there isn't much here that's earth-shattering, nor are there simple extractable principles that will make your experience designs better. As always, the takeaway boils down to "invest time and resources into this and try to think about it systematically," which of course is true of every possible axis of improvement for any product. (The challenge is usually where to spend those limited resources.) I will also note in passing that Lombardi assumes an entirely market-driven metric for success, and all of the case studies are exclusively commercial not unexpected, since that's where most of the money and a lot of the resources are, but not that helpful for bespoke work where the nature of the problem is subtlely different. But, despite the expected shortcomings and the inherent weakness of the case study approach to analysis, which tries to pluralize anecdote to get data I enjoyed this book. I also got at least one useful trick out of it: the premortem, to look for potential failure points in the experience before rolling out, or even designing, the product. It's a fairly light-weight book, but I think it's moderately inspirational, and I wholeheartedly concur with Lombardi on the merits of failure analysis as an engineering approach. Rating: 7 out of 10

Just saw this ad today at my Institute's door. Rough translation: Interim programmer sought Requisites:

• High school
• One year of experience
• Theorical and practical exams on the area
• Familiarity handling electronic equipment, such as computers, perforators/punchers, terminals and similar.

[Yes, this is a joke post. I don't know how anybody is using such an obsolete job description. A job description at least three decades out of date] [Well, yes, I understand it, because I knowhow that particular union works. But I cannot accept it is for real] [...And no, please don't apply for the job. You will be miserable in the unlikely event you are chosen]

Finally, more bootlog-pr0n:

start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
/dev/hd0s1: clean, 44680/181056 files, 292234/723200 blocks
done.
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Cleaning up temporary files... /tmp.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 8
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 34744 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
unexpected ACK from keyboard
GNU 0.3 (debian) (console)
login: root
root@debian:~# shutdown -h now
Broadcast message from root@debian (console) (Fri Aug 16 20:02:47 2013):
The system is going down for system halt NOW!
INIT: Switching to runlevel: 0root@debian:~#
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
Using makefile-style concurrent boot in runlevel 0.
Stopping deferred execution scheduler: atd.
task c10f72a8 deallocating an invalid port 2098928, most probably a bug.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds...done.
Stopping enhanced syslogd: rsyslogd.
Deconfiguring network interfaces...ifdown: failed to open pid file /run/network/ifdown-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
/bin/sh: 1: ifconfig: not found
done.
Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space
done.
Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons)
umount: could not find entry for: /dev/cons
done.
Unmounting local filesystems...done.
mount: cannot remount /: Device or resource busy
Will now halt.
init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)...
In tight loop: hit ctl-alt-del to reboot

(You might note that df complains about not being able to read the mtab file. That is because it has been built with _PATH_MOUNTED being /var/run/mtab. This will correct itself when the coreutils package is being rebuilt against a patched libc.) I spent my last two weeks with polishing my patch series, that meant a lot of package rebuilds and that means a lot of waiting (even more so on the Hurd, short version: fakeroot-hurd could be fast but is not yet working properly, fakeroot-tcp is slow) and thus some frustration ;) Also I had to pay special attention so that the upgraded packages could be installed without accidentally breaking anything in the process. Making sysvinit pid 1 is surprisingly tricky in this regard since it breaks the ABI and requires a libc fix that also worked with the current Hurd servers. Here are the patches:

• The mtab translator (v4) provides /proc/mounts: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html
• When I wrote umount I thought I implemented all the flags used by the initscripts package. Turns out I forgot -d. umount.patch (v2): http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00018.html
• Make sysvinit pid 1 (sysvinit kinda requires that), fix killall5 (that was freezing the system before) (v2): http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00020.html
• Making sysvinit pid 1 breaks the ABI, since the glibc assumes /hurd/init to have pid 1: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00046.html http://sourceware.org/ml/libc-alpha/2013-08/msg00205.html
• So now killall5 is not freezing the system anymore (actually it froze all processes but most importantly the procfs translator providing /proc and it then tried to walk over /proc which obviously blocked killall5 itself). Cool. But then it looks into /proc/*/stat for the fields start_code and end_code. So I dug into the exec server where the ELF decoding happens and handed this to the proc server, but the procfs translator has to make use of this: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00017.html
• Patch series for the sysvinit package (v2). This contains a runsystem.sysv that can be enabled using update-alternatives --config runsystem and several small fixes: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00038.html

So I had some spare time on my hand while waiting for numerous package rebuilds and I took this as an opportunity to read papers about Mach and to familiarize myself with mig, the Mach Interface Generator. While I have used it in the past, I had not yet looked at its implementation. And I had to patch the exec server, and there was both code implementing a questionable feature (on-demand unzipping of binaries) and code that was not even compiled (courtesy of the preprocessor) and had probably bit-rot by now. So I figured I could spend my time doing some cleanups:

• Cleanups for the exec server: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00049.html
• Cleanups for mig: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00055.html

I have rebuild all the necessary packages and uploaded them into an apt repository:

deb http://teythoon.cryptobitch.de/gsoc/heap/debian unstable main

Please use unstable for now. Also make sure that you have a recovery plan for your Debian/Hurd installation if anything goes wrong. For your convenience there's a seed tarball containing packages with the appropriate sources.list.d snippets and the repository key: https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar If you want to switch to the new runsystem.sysv, do:

# update-alternatives --config runsystem

Whenever you switch runsystems, please use reboot-hurd to reboot the system. This is the most robust way. Known issues:

• procfs hardcodes the default kernel pid to 2. This breaks /proc/uptime and any program relying on it, most notably top and friends. Until this is properly fixed, you can do:

  # settrans -apfg /proc /hurd/procfs -c -k 3
  

• The mtab translator should probably try to filter out non-filesystem translators. df complains loudly about /dev/cons for example.

Next week I will address the network related issues. By now they are the source of most of the error messages in the bootlog.

Review: Dark Lies the Island, by Kevin Barry

Publisher:	Greywolf
Copyright:	2012
Printing:	2013
ISBN:	1-55597-651-4
Format:	ARC
Pages:	185

This is a literary short story collection by a mainstream Irish writer, rather far from my normal fare. It was the second book in a shipment of Powell's Indiespensable, which is generally (when there is one) a proof or Advanced Reading Copy of something relatively obscure. I'm not really the target audience, not being much of a short story aficionado, but the whole point of doing this Indiespensable experiment is to stretch my reading range. First off, as I would generally expect by mainstream fiction, the quality of the writing is excellent. One thing I particularly like about Barry is that he's refreshingly and unhesitatingly profane. His characters curse like I expect people to curse, some of them at the drop of a hat, and for me it gives a feel of solid realism to his dialogue. He's also very good with eye dialect, giving the impression of accents without making the text hard to read. Some of these stories I liked a great deal, rather more than I had expected. But if there was a general undermining flaw for me, it was that I prefer stories that go somewhere, that have a plot or lead up to a definite point. Some of Barry's do, but a lot of them seem more vignettes or just scenes, sketches of characters, that drift off without resolution. I can admire those as writing exercises, but for me they don't fill the desire I have when I sit down with a story. I'm not sure how useful these reviews will be, given that I don't have a lot of mainstream fiction background to compare them with, but here are some reactions, for whatever they're worth. "Across the Rooftops": This is one of those sketches, but it's one of the best of them. It's a single moment on the rooftops, an attempt to capture the moment of possibility around the first romantic overture. It's sadly quite conventional in gender roles (nervous man initiating, woman as relationship gatekeeper), but apart from that it's very well-written. This kind of thing is normally not to my taste, but Barry does such a great job capturing the sense of uncertainty and the sharp focus on detail that I couldn't help but like it. (7) "Wifey Redux": This is by far my favorite story of this book. I think it's both hilarious and brilliant, right from the starting line.

This is the story of a happy marriage but before you throw up and turn the page let me say that it will end with my face pressed hard into the cold metal of the Volvo's bonnet, my hands cuffed behind my back, and my rights droned into my ear this will occur in the car park of a big-box retail unit on the Naas Road in Dublin.

It's the story of a marriage and how it changes over time, but also the story of being a father and his baffled, flailing reactions to his daughter's sexual explorations. But that's not what makes the story. What makes the story is what leads up to the arrest at the end, which had me laughing out-loud in delight. Barry is great at writing a sort of devilish humor; I wish more of the collection was like this. (9) "Fjord of Killary": This is another one of the ones I enjoyed. It's about an author who buys a hotel (with a bar) on the fjord of Killary thinking that the local culture and contact along with the wild beauty would inspire his poetry. At the start of the story, he's thoroughly sick of both the climate and the people, and the collision between his idyllic dreams and the reality, as well as his acerbic first-person complaints, is quite entertaining. A huge storm and resulting flood provides the conflict and impetus for change, and a few moments of humor in the understated local reactions. I thought the closing epiphany was a bit too obvious and expected, and the ending didn't quite work for me, but it has some good moments. (7) "A Cruelty": Here's where the collection started losing me. This is still well-written and closely observed, but in the case of this story, I wish it weren't, since I found it extremely uncomfortable. The viewpoint character is vulnerable to an encounter of unalloyed nastiness. I'm sure that's the point, but, similar to how I avoid horror, it's the sort of story I'd rather not read. (3) "Beer Trip to Llandudno": This was my second-favorite story of the collection. It's about a group of men who take periodic beer-tasting trips and rate beers. They're just normal men, from a variety of backgrounds and professions, and I thought Barry did a great job capturing the sort of camaraderie that comes from long association with a hobby. That includes, here, the awkward limits on what one talks about and the tendency to use the hobby as a shield from anything that drifts into less certain territory. I thought the story suffered somewhat from not having much of a plot or conclusion, but that's also necessary from the setup. This group of people isn't actually going to solve problems, but will be there for each other in their own way. (8) "Ernestine and Kit": Another very dark story, although that's not quite as obvious at the start. As with "A Cruelty," it's well-written but disturbing and lacks any sort of positive resolution. Here, I think Barry went a bit far into portraying people who are a large part of the popular imagination but who are exceptionally rare in reality. It's the kind of story that, at least for me, plays badly with my brain's threat analysis: too memorable for the level of actual risk, and feeling like it was playing a bit too much with popular boogeymen. (4) "The Mainland Campaign": This one was just too subtle for me. Some of the summaries for this book hint at a meaning for the story that I didn't pick up at all on my first reading. On re-reading, I suspect that analysis is right, but even re-reading the story with that interpretation in mind (this is the sort of story where I'm not sure the average American would pick it up on first reading, but others might), I still don't understand exactly what happened. I think some mainstream writing, and some short stories, tends to err on the side of subtle. (4) "Wistful England": Another one that's way over on the sketch of a moment side of stories. Unlike "Across the Rooftops," it failed to make any impression on me whatsoever; I had to skim it just now to remind myself of any details at all, and by the time I edited this review for posting, I'd forgotten it again. I think it was aiming at capturing a mood, but the mopey viewpoint character didn't mean anything to me. (3) "Doctor Sot": This is another odd one, and I'm not quite sure what I think of it. Either it's another character sketch without much of a plot, or the plot was too subtle. The protagonist is a rather unlikable drunk who doubles as a small town's incompetent doctor. The story involves his encounter with a local sort of hippie encampment, which is moderately interesting even though I couldn't stand the man. Then it leads to an encounter that clearly meant something, but which was entirely lost on me. (4) "The Girls and the Dogs": And from an absence of plot back to the deeply disturbing. This story is about a moderately unlikable drug dealer who gets refuge from an extremely unlikable crazy person in a weird relationship with a woman and her sister. (As is sadly typical, polyamory only makes an appearance when the author wants to paint a scenario that's utterly fucked up and psychotic.) There's a pseudo-fantasy element here in that the characters talk about spells, but I think the only spell involved is authorial fiat to create a deeply broken and sick game that plays into a host of negative stereotypes. Despite myself, I did get drawn into the suspense of the story (Barry's undoubtedly a good writer), but it left me feeling dirty on several levels. (4) "White Hitachi": Finally, another story that I liked, although not as good as the ones earlier in the collection. This one follows a con man and two-bit thief as he gets his brother out of lockup and tries to resolve, or at least stay ahead of, various problems in his life. Again, no real ending, but I thought Barry did a good job at taking a snapshot of a life from a perspective that I don't read much in literature (along with all the sexism and racism that comes along with it). It sort of drifts off rather than ends (although the last sentence is a great bit of characterization), but I found it oddly enjoyable. (6) "Dark Lies the Island": I'm not sure what to make of this. It's another character sketch, a moment of deep psychological significance for a character, floating somewhat alone but described in detail. But the character is a girl struggling with cutting, from a male author whose other stories did not fill me with confidence in his handling of female characters. Barry makes this psychologically believable, but I'm a little nervous about trusting his portrayal of that mindset. It's also, again, just a bit too subtle; I had a hard time getting an emotional feel for how the events of the story connect. (5) "Berlin Arkonaplatz My Lesbian Summer": The final story of the collection is fairly typical of the problem I had with most of it. It seems reasonably well-written, I had a hard time connecting with the protagonist or understanding his motivations. It looks at a slice of life with which I have no familiarity at all (and is interesting for that reason), but I don't quite trust the story enough to submerge in it. Barry seems to write a lot of protagonists who seem aimless, befuddled, or just largely out of control, and I like a bit more agency in the primary character. Silvija is an interesting character, even shown from another's perspective, but the story is almost purely observational: showing her without comment, without obvious motive, and mostly without plot. It's strangely compelling, but not quite compelling enough that I'd actually seek it out. Which, really, is the whole book in a nutshell. (6) Rating: 6 out of 10

I found the following news item; if you can read Spanish, you will most probably prefer the original version in the Proceso magazine's site. The subject? The federal police (PGR) and army arrest 17 artisans for making money out of Spiderman. The following translation is mine. Done past midnight, and being quite tired, and translated so this news item can reach a broader audience. All errors are mine (except those carried out by the security forces, that is).

June 13, 2013 Cuernavaca, Morelos. Policement from the General Republic Attorney (Procuradur a General de la Rep blica, PGR) and the Army entered and searched the "3 de mayo" neighbourhood, in the municipality of Emiliano Zapata, detaining 17 ceramist artisans that sold candies, dolls and pi atas shaped like Spiderman. This search was done on the evening of last Wednesday, around 16:00. Federal ministerial policement and army soldiers closed a street with several informal stores and detained workers taht were selling this Marvel Comics character, following said company's denounce. As a result for this operation, 17 artisants were detained, although the same day five of them were freed. The policemen also seized 12 bags of candies, pi atas, ceramics and wooden figures of the superhero. PGR closed down 11 stores where ceramics with this same figure was being sold, accusing the detainees of plagiarizing Spiderman's image, protected under the copyright law. The 12 that remained under detention were put at the Federal Justice's disposal, which prompted that this Thursday, around 10AM, hundreds of sellers of "3 de mayo" went out to PGR's building to demand their friends' freedom, who are facing a bail of up to 200,000 pesos (~USD$18,000). Outraged because they said they were treated as if they were part of a drug ring, hundreds of artisans closed intermitently Avenida Cuauhn huac, where the PGR representation in Morelos state is located. The artisans' pressure helped for the amount of the bail to be lowered from MX$200,000 to MX$16,000, and so they were set free. Francisco Fern ndez Flores, president of the Ceramists Association, criticized the operation because, he said, it was as strong as if they were "drug dealers". The artisans explained that they don't even make the Spiderman figures, they are made by the interns of the Centro Estatal de Reinserci n Social de Atlacholoaya (prision), located in the Xochitepec municipality, who offered them to the ceramists so they could be sold. "The Atlacholoaya inmates do them, we buy them to support them, and turns out we are the delinquents now", said Miriam Monroy, sister of one of the detainees. This information was contradicted by Jes s Valencia Valencia, responsible for Morelos' state prision system, who assured that in said prision no ceramics are done. Fern ndez Flores insisted though that from within the prision they are being offered pi atas, candies and "piggy banks" with Spiderman's shape. Jos Luis Pozo, vicepresident of the Ceramists Union, said that to avoid more such federal operations for copyright breaches, they have committed not to produce or commercialize Marvel superhero figures, and any other characters the authority demands. "We do commit to, from now on, those products singled out to us will not be commercialized", he said. Pozo said that the PGR operation caused losses not just to the detained producers and salesmen, but to over 200 ceramists that had to close their stores in solidarity with their friends. Acording to the artisans, the products were a success until the PGR came, seized the products and detained the salesmen.

And yes, the copyright insanity does not stop. Spiderman is by today a clear part of popular culture. Marvel brilliantly succeeded in creating such a popular icon that everybody recognizes, that everybody identifies with And that everybody should be able to recreate. We are not talking about brand protection. Marvel does not, and will never, commercialize pi atas, ceramics or wooden toys. And even if they were plastic-cast While Spiderman is still under the protection of copyright, as the Berne Convention defines it (and of course, as the much stricter Mexican laws agree), that does not mean that any and every product resembling a Spiderman should be protected. Many ceramists and pi ata makers will create unique pieces of art Ok, handicraft. But reading the copyright law more strictly, Spiderman is more treated as a trademark than as a copyright. And it is a trademark that should be declared as having passed on to the public domain.

CALL FOR PARTICIPATION: Shogun Machine Learning Workshop, Berlin, Germany, July 12-14, 2013 Data Science, Big-Data are omnipresent terms documenting the need for automated tools to analyze the ever growing wealth of data. To this end we invite practitioners, researchers and students to participate in the first Shogun machine learning workshop. While the workshop is centered around the development and use of the shogun machine learning toolbox, it will also feature general machine learning subjects. General Information The workshop will include:

• A general introduction to machine learning held by Gunnar Raetsch.
• Introductory talks about e.g. Dimension reduction techniques, Kernel-statistical testing, Gaussian Processes, Structured Output learning.
• Contributed talks and a poster session, and a poster-spotlight.
• A discussion panel
• A hands on session on July 13-14

Do not miss the chance to familiarize yourself with the shogun machine learning toolbox for solving various data analysis tasks and to talk to their authors and contributors. The program of the workshop will cover from basic to advanced topics in machine learning and how to approach them using Shogun, which makes it suitable for anyone, no matter if you are a senior researcher or practitioner with many year's of experience, or a junior student willing to discover much more. Interested? A tentative schedule is available at http://shogun-toolbox.org/page/Events/workshop2013_program. Call for contributions The organizing committee is seeking workshop contributions. The committee will select several submitted contributions for 15-minute talks and poster presentations. The accepted contributions will also be published on the workshop web site. Amongst other topics, we encourage submission that

• are applications / publications utilizing Shogun
• are highly relevant to practitioners in the field
• are of broad general interest
• are extensions to Shogun

Submission Guidelines Send an abstract of your talk/contribution to shogun-workshop2013@shogun-toolbox.org before June 1. Notifications will be given on June 7. Registration Workshop registration is free of charge. However, only a limited number of seats is available. First-come, first-served! Register by filling out the registration form. Location and Timeline The main workshop will take place at c-base Berlin (http://c-base.org/, https://en.wikipedia.org/wiki/C-base) on July 12. It is followed by additional 2-day hands-on sessions held at TU Berlin on July 13-14. About the Shogun machine learning toolbox Shogun is designed for unified large-scale learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. Further information is available at http://www.shogun-toolbox.org.

Subunit is seven and a half years old now Conrad Parker and I first sketched it up at a CodeCon camping and coding, a brilliant combination in mid 2005.

revno: 1
committer: Robert Collins <robertc@robertcollins.net>
timestamp: Sat 2005-08-27 15:01:20 +1000
message:  design up a protocol with kfish

It has proved remarkably resilient as a protocol the basic nature hasn t changed at all, even though we ve added tags, timestamps, support for attachments of arbitrary size. However a growing number of irritations have been building up with it. I think it is time to design another iteration of the protocol, one that will retain the positive qualities of the current protocol, while helping it become suitable for the next 7 years. Ideally we can keep compatibility and make it possible for a single stream to be represented in any format. Existing design The existing design is a mostly human readable line orientated protocol that can be sniffed out from the regular output of make or other build systems. Binary attachments are done using HTTP chunking, and the parser has to maintain state about the current test, tags, timing data and test progression [a simple stack of progress counters]. How to arrange subunit output is undefined, how to select tests to run is undefined. This makes writing a parser quite easy, and the tagging and timestamp facility allow multiplexing streams from two or more concurrent test runs into one with good fidelity but also requires that state be buffered until the end of a test, as two tests cannot be executing at once. Dealing with debuggers The initial protocol was intended to support dropping into a debugger just pass each line read through to stdout, and connect stdin to the test process, and voila, you have a working debugger connection. This works, but the current line based parsers make using it tedious the line buffered nature of it makes feedback on what has been typed fiddly, and stdout tends to be buffered, leading to an inability to see print statements and the like. All in-principle fixable, right ? When running two or more test processes, which test process should stdin be connected to? What if two or more drop into a debugger at once? What is being typed to which process is more luck than anything else. We ve added some idioms in testrepository that control test execution by a similar but different format one test per line to list tests, and have runners permit listing and selecting by a list. This works well, but the inconsistency with subunit itself is a little annoying you need two parsers, and two output formats. Good points The current protocol is extremely easy to implement for emitters, and the arbitrary attachments and tagging features have worked extremely well. There is a comprehensive Python parser which maps everything into Python unittest API calls (an extended version of the standard, with good backwards compatibility). Pain points The debugging support was a total failure, and the way the parser depraminates it s toys when a test process corrupts an outcome line is extremely frustrating. (other tests execute but the parser sees them as non-subunit chatter and passes the lines on through stdout). Dealing with concurrency The original design didn t cater for concurrency. There are three concurrency issues the corruption issue (see below for more detail) and multiplexing. Consider two levels of nested concurrency: A supervisor process such as testrepository starts 2 (or more but 2 is sufficient to reason about the issue) subsidiary worker processes (I1 and I2), each of which starts 2 subsidiary processes of their own (W1, W2, W3, W4). Each of the 4 leaf processes is outputting subunit which gets multiplexed in the 2 intermediary processes, and then again in the supervisor. Why would there be two layers? A concrete example is using testrepository to coordinate test runs on multiple machines at once, with each machine running a local testrepository to broker tests amongst the local CPUs. This could be done with 4 separate ssh sessions and no intermediaries, but that only removes a fraction of the issues. What issues? Well, consider some stdout chatter that W1 outputs. That will get passed to I1 and from there to the supervisor and captured. But there is nothing marking the chatter as belonging to W1: there is no way to tell where it came from. If W1 happened to fail, and there was a diagnostic message printed, we ve lost information. Or at best muddled it all up. Secondly, buffering imagine that a test on W1 hangs. I1 will know that W1 is running a test, but has no way to tell the supervisor (and thus the user) that this is the case, without writing to stdout [and causing a *lot* of noise if that happens a lot]. We could have I1 write to stdout only if W1 s test is taking more than 5 seconds or something but this is a workaround for a limitation of the protocol. Adding to the confusion, the clock on W1 and W3 may be very skewed, so timestamps for everything have to be carefully synchronised by the multiplexer. Thirdly, scheduling if W1/W2 are on a faster machine than W3/W4 then a partition of equal-timed tests onto each machine will lead one idle before the other finishes. It would be nice to be able to pass tests to run to the faster machine when it goes idle, rather than having to start a new runner each time. Lastly, what to do when W1 and W2 both wait for user input on stdin (e.g. passphrases, debugger input, $other). Naively connecting stdin to all processes doesn t work well. A GUI supervisor could connect a separate fd to each of I1 and I2, but that doesn t help when it is W1 and W2 reading from stdin. So additional requirement over baseline subunit:

1. make it possible for stdout and stderr output to be captured from W1 and routed through I1 to the supervisor without losing its origin. It might be chatter from a noisy test, or it might be build output. Either way, the user probably will benefit if we can capture it and show it to them later when they review the test run. The supervisor should probably show it immediately as well the protocol doesn t need to care about that, just make it possible.
2. make it possible to pass information about tests that have not completed through one subunit stream while they are still incomplete.
3. make it possible (but optional) to pass tests to run to a running process that accepts subunit.
4. make it possible to route stdin to a specific currently process like W1. This and point 3 suggest that we need a bidirectional protocol rather than the solely unidirectional protocol we have today. I don t know of a reliable portable way to tell when some process is seeking such input, so that will be up to the user I think. (e.g. printing (pdb) to stdout might be a sufficiently good indicator.)

Dealing with corruption Consider the following subunit fragment:

test: foo
starting serversuccess:foo

This is a classic example of corruption: the test foo started a server and helpfully wrote to stdout explaining that it did that, but missed the newline. As a result the success message for the test wasn t printed on a line of its own, and the subunit parser will believe that foo never completed. Every subsequent test is then ignored. This is usually easy to identify and fix, but its a head-scratcher when it happens. Another way it can happen is when a build tool like make runs tests in parallel, and they output subunit onto the same stdout file handle. A third way is when a build tool like make runs two separate test scripts serially, and the first one starts a test but errors hard and doesn t finish it. That looks like:

test: foo
test: bar
success: bar

One way that this sort of corruption can be mitigated is to put subunit on it s own file descriptor, but this has several caveats: it is harder to tunnel through things like ssh and it doesn t solve the failing test script case. I think it is unreasonable to require a protocol where arbitrary interleaving of bytes between different test runner streams will work so the make -j2 case can be ignored at the wire level though we should create a simple way to safely mux the output from such tests when the execute. The root of the issue is that a dropped update leaves bad state in the parser and it never recovers. So some way to recover, or less state to carry in the parser, would neatly solve things. I favour reducing parser state as that should shift stateful complexity onto end nodes / complex processors, rather than being carried by every node in the transmission path. Dependencies Various suggestions have been made JSON, Protobufs, etc A key design goal of the first subunit was a low barrier to entry. We keep that by being backward compatible, but being easy to work with for the new revision is also a worthy goal. High level proposal A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag one of (estimate + number, stdin, stdout, stderr, test- + test id), test status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes. The content tags:

• estimate the stream is reporting how many tests are expected to run. It affects everything with the same routing code only, and replaces (doesn t adjust) any current estimate for that routing code. A estimate packet of 0 can be used to say that a routing target has shutdown and cannot run more tests. Routing codes can be used by a subunit aware runner to separate out separate threads in a single process, or even just separate TestSuite objects within a single test run (though doing so means that they will need to process subunit and strip packets on stdin. This supercedes the stack of progress indicators that current subunit has. estimates cannot have test status or attachments.
• stdin/stdout/stderr: a packet of data for one of these streams. The routing code identifies the test process that the data came from/should go to in the tree of test workers. These packets cannot have test status but should have a non-empty attachment block.
• test- + testid: a packet of data for a single test. test status may be included, as may attachment name, mime type, last-block and binary data.

Test status values are pretty ordinary. Exists is used to indicate a test that can be run when listing tests, and inprogress is used to report a test that has started but not necessarily completed. Attachment names must be unique per routing code + testid. So how does this line up? Interleaving and recovery We could dispense with interleaving and say the streams are wholly binary, or we can say that packets can start either after a \n or directly after another packet. If we say that binary-only is the approach to take, it would be possible to write a filter that would apply the newline heuristic (or even look for packet headers at every byte offset. I think mandating adjacent to a packet or after \n is a small concession to make and will avoid tools like testrepository forcing users to always configure a heuristic filter. Non-subunit content can be wrapped in subunit for forwarding (the I1 in W1->I1->Supervisor chain would do the wrapping). This won t eliminate corruption but it will localise it and permit the stream to recover: the test that was corrupted will show up as incomplete, or with incomplete attachment data. listing Test listing would emit many small non-timestamped packets. It may be useful to have a wrapper packet for bulk amounts of fine grained data like listing is, or for multiplexers with many input streams that will often have multiple data packets available to write at once. Selecting tests to run Same as for listing while passing regexes down to the test runner to select groups of tests is a valid use case, thats not something subunit needs to worry about : if the selection is not the result of the supervisor selecting by test id, then it is known at the start of the test run and can just be a command line parameter to the backend : subunit is relevant for passing instructions to a runner mid-execution. Because the supervisor cannot just hand out some tests and wait for the thing it ran to report that it can accept incremental tests on stdin, supervisor processes will need to be informed about that out of band. Debugging Debugging is straight forward . The parser can read the first 4 or so bytes of a packet one at a time to determine if it is a packet or a line of stdout/stderr, and then either read to end of line, or the binary length of the packet. So, we combine a few things; non-subunit output should be wrapped and presented to the user. Subunit that is being multiplexed and forwarded should prepend a routing code to the packet (e.g. I1 would append 1 or 2 to select which of W1/W2 the content came from, and then forward the packet. S would append 1 or 2 to indicate I1/I2 the routing code is a path through the tree of forwarding processes). The UI the user is using needs to supply some means to switch where stdin is attached. And stdin input should be routed via stdin packets. When there is no routing code left, the packet should be entirely unwrapped and presented as raw bytes to the process in question. Multiplexing Very straight forward unwrap the outer layer of the packet, add or adjust the routing code, serialise a header + adjusted length + rest of packet as-is. No buffering is needed, so the supervisor can show in-progress tests (and how long they have been running for). Parsing / representation in Python or other languages The parser should be very simple to write. Once parsed, this will be fundamentally different to the existing Python TestCase->TestResult API that is in used today. However it should be easy to write two adapters: old-style <-> this new-style. old-style -> new-style is useful for running existing tests suites and generating subunit, because thats way the subunit generation is transparent. new-style->old-style is useful for using existing test reporting facilities (such as junitxml or html TestResult objects) with subunit streams. Importantly though, a new TestResult style that supports the features of this protocol would enable some useful features for regular Python test suites:

• Concurrent tests (e.g. in multiprocessing) wouldn t need multiplexers and special adapters a regular single testresult with a simple mutex around it would be able to handle concurrent execution of tests, and show hung tests etc.
• The routing of input to a particular debugger instance also applies to a simple python process running tests via multiprocessing, so the routing feature would help there.
• The listing facility and incrementally running tests would be useful too I think we could go to running tests concurrently with test collection happening, but this would apply to other parts of unittest than just the TestResult

The API might be something like:

class StreamingResult(object):
    def startTestRun(self):
        pass
    def stopTestRun(self):
        pass
    def estimate(self, count, route_code=None):
        pass
    def stdin(self, bytes, route_code=None):
        pass
    def stdout(self, bytes, route_code=None):
        pass
    def test(self, test_id, status, attachment_name=None, attachment_mime=None, attachment_eof=None, attachment_bytes=None):
        pass

This would support just-in-time debugging by wiring up pdb to the stdin/stdout handlers of the result object, rather than actual stdin/stdout of the process a simple matter once written. Alternatively, the test runner could replace sys.stdin/stdout etc with thunk file-like objects, which might be a good idea anyway to capture spurious output happening during a test run. That would permit pdb to Just Work (even if the test process is internally running concurrent tests.. until it has two pdb objects running concurrently Generation new streams Should be very easy in anything except shell. For shell, we can have a command line tool that when invoked outputs a subunit stream for one instruction. E.g. test foo completed + some attachments or test foo starting .

Earlier this week, things were looking different, and I did have to cancel a Haskell talk in Freiburg on Monday due to illness. But I m back on track and will be travelling to Brussels tomorrow. I ll be holding a talk on how we package Haskell in Debian, on Suday at 15:30. I hope it will be useful to Debian users (who will better understand the packaging), other Debian Developers (who ll learn about the peculiarities of Haskell and the implications for the Debian infrastructure), other distro s maintainers (to compare best practices) and Haskell developers (to learn about the needs and worries of downstream packages). The talk will be based on my DebConf 11 talk on the same topic. I m also happy to answer questions about Haskell, Haskell in Debian or any other topic that you want to hear my opinion about, so just talk to me during FOSDEM.
In related news: GHC 7.6.2 was uploaded to Debian experimental the day it was released; the rebuilding of all libraries is still in progress (~370 of ~570 done).

This is one of the days where reading my everyday newspaper was worth more than just getting bitter at the news. I found this text in La Jornada, my usual newspaper. I liked it very much, and decided to translate it for a wider audience. Of course, if you can read Spanish, do yourself a favor and go to the original. It is not that the text is so easy to translate. And, after all, I'm not a native English writer. I'm trying to do a literal translation, even when disagreeing with the author. Ten theses in favor of free download of cultural goods on the Internet Enrique G. Gallegos Poet and philosopher. Currently a researcher in Universidad Aut noma Metropolitana-C

1. Historical legacy. Criticists who promote the persecution of piracy and free downloads of cultural works from the Internet argue that the patrimonial rights have been wronged; they start off suposing that a cultural work is formed from a historical void, as if there was a "nothing" to begin with, and then a "something" appeared. Nothing more fallacious: All cultural products has its precedents, and thanks to these it generates part of its best forces.
2. Opening towards the future. Human beings are projected into the future. As the main philosophies of the 20th century have explored, one of the singular characteristics of mankind is the ability to think and imagine the future. Products of culture are the best ways to think and project society, politics, love, hope, needs, failures A song, a poem, an essay, can trigger imaginary worlds with a transformative potential. To deny this to humanity with "patrimonial damage" as an argument is to mutilate man's temporal nature.
3. Recognition of the present. Cultural works generate moments of reflection, criticism and pleasure; they can trigger actions demanding commitment, solidarity and strong indignation against injustice. If every cultural work is adscribed in some way to a historic tradition, and if cultural products anre necessary to imagine other possible worlds and open our future, it's not less true that they also allow us to recognize our present by making us more sensitive to others' ideas, sensations and emotions. Without culture, there is no present nor others' presence.
4. Divulgation of the cultural heritage. If culture is "heritage of mankind", it should be also spread by any possible way. But not just spread, but actively sought for the biggest number of people to be able to effectively have access to it. In a potentially interconnected world, Internet is the most adequate world for it (despite its limitations). Therefore, penalizing free downloads means avoiding spreading it and restricting access to those groups who don't have enough economic resources.
5. Preserving cultural heritage. If we agree that cultural works are heritage of mankind, we need to take every action needed to preserve them. But preserving culture does not only mean keeping the products in museums, galleries or never-polluted drawers; it rather means to keep them in the collective memory and in the flow of constant interpretations and appropiations. Said in other words, the genuine way to preserve culture is to allow universal access to cultural goods. That's why, holding that free copies and downloads of books, music, videos, etc. over the Internet are harmful, is an incompatible argument with the obligation to preserve cultural heritage.
6. They are not goods. Cultural works are material and spiritual manifestations of mankind, concretions of its historicity; they are, also, expressions of emotions, ideas and projections of other lives and other worlds. That's why they cannot be equated to goods or inserted in patrimonialist logic. The market might want to cheat by giving a certain price to a painting by Orozco or auctioning a Baudelaire manuscript, but it will never be able to aprehend their true meaning as cultural works. And it's not that culture is not faluable, but taht its estimation criteria do not follow the rules of market, but those of the imponderable and unlimited.
7. Price disproportion. Assuming we accept the possibility that intermediaries charge for the services they perform, the value of a book, record or movie should never exceed the daily wage of a worker or employee. But this ellection would only be one more option within the effective possibility that people opt to download or freely copy the cultural work. The final decision as to which media to go to should be a sovereign resolution of the culture-interested person.
8. The principle of the most benefit. Even when copies and free downloads over Internet could generate a patrimonial "wrong" to third parties, the cultural benefit obtained by it will always be greater in as much that it carries out intensely the principle to foster, spread and access the cultural heritage of mankind. To think otherwise is to privilege the few over the many.
9. The real evil is elsewhere. Murder, women- and children-trade, poverty and misery are the real evils that ache mankind. According to the United Nations, in 2010 there were 468 thousand murders in the world; the estimation is that 3500 million people live in poverty; in their 2009 inform, the UN found over 2400 victims of "people trade", kidnapped as sex slaves; in some regions in Africa, close to 30% of children suffer acute malnutrition and six people (between children and adults) die every day from hunger. That's why pretending that free download of cultural works is an evil is clearly a tricky and deceptive speech
10. Against the intermediary-salesman. Culture does not need intermedieries that reduce cultural works to goods. Culture is too important to leavie it in the hands of salespeople that simplistly equate cultural goods with cakes. An intermediary of this kind will never understand the difference between a work of art and a disposable razor blade. What the world needs is more support from the government from every country to artists, creators and poets, as well as ensuring the conditions for the absolute mobility freedom for cultural works.

Date: Mon, 18 Mar 2002 18:22:10 +0000
From: James Troup <troup@samosa.debian.org>
To: Aurelien Jarno <aurelien@aurel32.net>
Cc: da-manager@debian.org
Subject: New Debian maintainer Aurelien Jarno [ This is a long (automatically-generated) mail, but it contains
important information, please read it all carefully. ] Dear Aurelien Jarno! An account has been created for you on developer-accessible machines with username aurel32 . The password for this account can be found encrypted with your PGP or GPG key and appended to this message. A list of machines available to Debian developers can be found at <URL:http://db.debian.org/machines.cgi>. Please take a minute now to familiarize yourself with the Debian Machine Usage Policy, available at <URL:http://www.debian.org/devel/dmup> You have been subscribed to the debian-private mailing list as <aurel32@debian.org>. Please respect the privacy of that list and don t forward mail from it elsewhere. E-mail to <aurel32@debian.org> will be forwarded to <aurelien@aurel32.net>. To change this, please see <URL:http://db.debian.org/forward.html> Also, please subscribe to debian-devel-announce, if you haven t done so already. We strongly suggest that you use your aurel32@debian.org address for the maintainer field in your packages, because that one will be valid as long as you are a Debian developer, even if you change jobs, leave university or change Internet Service providers. If you do so, please add that address to your PGP/GPG key(s) (using gpg edit-key YOUR USER ID ) and send it to the keyring server at keyring.debian.org with gpg keyserver keyring.debian.org send-keys YOUR USER ID . You can find more information useful to developers at <URL:http://www.debian.org/devel/> (in particular, see the subsection titled Debian Developer s reference ). We suggest that you subscribe to debian-mentors@lists.debian.org. This list is for new maintainers who seek help with initial packaging and other developer-related issues. Those who prefer one-on-one help can also post to the list, and an experienced developer may volunteer to help you. You can get online help on IRC, too, if you join the channel #debian-devel on irc.debian.org. Take a look at the support section on www.debian.org in order to find out more information. You should have read these documents before working on your packages. o The Debian Social Contract
<URL:http://www.debian.org/social_contract.html> o The Debian Policy Manual
<URL:http://www.debian.org/doc/debian-policy/> If you have some spare time and want to contribute it to Debian you may wish to take a look at the Work-Needing and Prospective Packages for Debian GNU/Linux also known as WNPP that can be found at <URL:http://www.debian.org/devel/wnpp/> If you plan to make a Debian package from a not yet packaged piece of software you *must* announce your intention on the debian-devel mailing list to make sure nobody else is working on them. The machine ftp-master.debian.org is our main archive server. Every uploaded package finds it s way there (except for Packages covered by US crypto laws which go to non-us.debian.org) eventually. master.debian.org is the home of our bug tracking system. Project web pages and CVS archives are hosted on klecker.debian.org (aka cvs/www.debian.org), klecker is also our general shell server. Web pages should be placed in public_html on klecker and refered to by http://people.debian.org/~aurel32 You should use ssh to log into the machines instead of regular telnet or rlogin. Our LDAP directory is able to share ssh RSA keys among machines, please see <URL:http://db.debian.org/doc-mail.html> Otherwise when you first login a ~/.ssh directory will be created with the appropriate permissions. Please be aware of the security implications of using RSA authentication and ssh agents. Finally, please take a minute to visit <URL:http://db.debian.org/>.
Login using the password information appended to this email, and update your personal information. The information is used to maintain your accounts on various Debian machines, and also to allow other developers and general users to find out more about you. Many of the fields are only visible to other registered Debian developers. This is also the only way to change your password. The passwd program does not yet work. Welcome to the project!
The Debian New Maintainer Team

git-annex has special remotes that allow large files checked into git to be stored in arbitrary places, that are not proper git remotes. One key use of the special remotes is to store files in The Cloud. Until now the flagship special remote used Amazon S3, although a few other things like Archive.org, rsync.net, and Tahoe-Laffs can be made to work too. One of my goals is to add as many cloud storage options to git-annex as possible. Box.com came to my attention because they currently have a promotion that provides 50 gigabytes of free "lifetime" service. Which is a nice amount of cloud storage to have for free. I decided that I didn't want to spend more than 4 hours of my time to make git-annex use it though. (I probably have spent a week on the S3 support by contrast.) So, this is a case study in quickly adding support for one cloud storage provider to git-annex.

• First, I had to sign up to box.com. Their promotion requires an android phone be used to get the 50 gigabytres. This wasted about an hour getting my unused phone dusted off etc. This also includes time spent researching ways to access box.com's storage, including reading their API documentation. I found it has a WebDAV interface.
• Sadly, there is not yet a native WebDAV library for haskell. This is a shame, because it would make the implementation better. But, I'm confident someone will eventually write one. My experience with haskell libraries for other web APIs (S3, GitHub) is that it's an excellent language to write them in, the code tends to be very simple, concise and clear. But I can't do it in 4 hours. So for now, the workaround is to use a WebDAV mounting tool. I picked davfs2 as it was the first one I got to work with box.com's slightly broken WebDAV. 2 hours spent now.
• With box.com mounted, I was neary done; git-annex's directory special remote can use the mount point. But there was a catch: box.com only allows up to 100 mb large files. I spent 1 hour or so adding support to the directory special remote for chunking files into a user-specified size.
  This was a fairly complex problem -- the existing code had a ByteString that when accessed lazily read the whole large file (from disk or from gpg, depending), and just called writeFile on it.
  I needed to still consume it lazily to avoid reading the whole file into memory, but write out chunks. This gets a bit into haskell's ByteString internals, but they're very well suited to this kind of thing, and so after 15 minutes familiarizing myself with the data structures, it was actually fairly easy to write the code. patch
• I spent my last hour testing and tuning the box.com special remote. Using davfs2 as a quick fix caused some technical debt that I had to make up for. In particular, the chunked filename retrieval code had to make sure not to open every chunk at once, because that makes davfs2 try to cache them all, instead of streaming one at a time. patch
• Not counted toward my 4 hour limit is the ... er ... 4 hours I spent last night adding a progress bar to the directory special remote. A progress display while transferring the files makes using box.com as a special remote much nicer, but also makes using my phone's SD card as a special remote much nicer! This is why I'm a poor consultant -- when faced with something generic and generally useful like this, I have difficulty billing for it.

The end result is that there are detailed instructions for using box.com as a special remote. And it seems to work quite well now. I just set up my production box.com special remote. All content written to it is gpg encrypted, and various of my computers have access to it, each using their own gpg key to decrypt the files uploaded by the others. (git-annex's encryption feature makes this work really well!) So..
There is a DropBox API for haskell. But as I'm not a customer, the 2 gb free account hardly makes it worth my while to make git-annex use it. Would someone like to fund my time to add a dropbox special remote to git-annex?

Last Friday, after 5 long years, I have finally submitted my PhD thesis. It was quite a relief, more or less as if an elephant was taken off my back. An English title for my thesis would be Structural Complexity Characterization in Software Systems. Here is an abstract:

This thesis proposes a theory to characterize structural complexity in software systems. This theory aims to identify (i) the contribution of several factors to the structural complexity variation and (ii) the effects of structural complexity in software projects. Possible factors in the structural complexity variation include: human factors, such as general experience of the developers and their familiarity with the different parts of the system; factors related to the changes performed on the system, such as size variation and change diffusion; and organizational factors, such as the maturity of the software development process and the communication structure of the project. Effects of structural complexity include higher effort, and consequently higher cost, in software comprehension and maintenance activities. To test the validity of the proposed theory, four empirical studies were performed, mining data from free software project repositories. We analyzed historical data from changes performed in 13 systems from different application domains and written in different programming languages. The results of these studies indicated that all the factors studied influenced the structural complexity variation significantly in at least one of the projects, but different projects were influenced by different sets of factors. The models obtained were capable of describing up to 93% of the structural complexity variation in the projects analyzed. Keywords: Structural Complexity, Software Maintainance, Human factors in Software Engineering, Mining Software Repositories, Theories in Software Engineering, Empirical Software Engineering, Free/Open Source Software Projects.

Those who read Portuguese can check out the actual thesis text as a PDF file. Most of the studies discussed in the thesis are presented in English in papers I have published during the last years. My defense is going to be on March 23rd. If you happen to be at Salvador at that day, please feel cordially invited.

I have been in the habit of using R to make e.g. histograms of test scores in my courses. The main problem is that I don't really need (or am too ignorant to know that I need) the vast statistical powers of R, and I use it rarely enough that its always a bit of a struggle to get the plot I want. racket is a programming language in the scheme family, distinguished from some of its more spartan cousins by its "batteries included" attitude. I recently stumbled upon the PLoT graph (information visualization kind, not networks) plotting module and was pretty impressed with the Snazzy 3D Pictures. So this time I decided try using PLoT for my chores. It worked out pretty well; of course I am not very ambitious. Compared to using R, I had to do a bit more work in data preparation, but it was faster to write the Racket than to get R to do the work for me (again, probably a matter of relative familiarity).

#lang racket/base
(require racket/list)
(require plot)
(define marks (build-list 30 (lambda (n) (random 25))))
(define out-of 25)
(define breaks '((0  9) (10 12) (13 15) (16 18) (19 21) (22 25)))
(define (per-cent n)
  (ceiling (* 100 (/ n out-of))))
(define (label l)
  (format "~a-~a" (per-cent (first l)) (per-cent (second l))))
(define (buckets l)
  (let ((sorted (sort l <)))
    (for/list ([b breaks])
          (vector (label b)
           (count (lambda (x) (and 
                    (<= x ( second b))
                    (>= x ( first b))))
               marks)))))
(plot
 (list
  (discrete-histogram 
  (buckets marks)))
 #:out-file "racket-hist.png")